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ARTICLE INFO ABSTRACT 


Keywords: We conducted an exploratory study on affect regulation during game-based learning where 110 college-aged 
Affect participants (Mage = 22.14, SDage = 1.24; 50.0% female; 70.0% White) played an easy, medium, and difficult 
Frustration level of an educational game (Physics Playground) while self-reporting their strongest affective state and reg- 
Affective regulation ulation strategies associated with each level. Participants also self-reported their effort and completed a physics 
peat ein - posttest after gameplay. We found that frustration, confusion, determination, and curiosity were the dominant 
Game-based learning affective states (81.4% of total reports) while cognitive reappraisal and acceptance were the major affective 
regulation strategies (others individually occurred less than 10.1% of the time). Engaging in cognitive re- 
appraisal — an affective regulation strategy that involves changing the way one thinks about a situation — was 
beneficial for successfully solving a level when participants were frustrated or confused, but had no effect when 
participants were determined or curious. Engaging in cognitive reappraisal when experiencing high frustration/ 
confusion positively predicted posttest scores, but only for those who put a high amount of effort into the game. 
For students who were low in effort or low in frustration/confusion, simply accepting one's emotions when 
experiencing high frustration/confusion was beneficial. We discuss theoretical implications and applications 


towards game-based learning supports to promote persistence and learning outcomes. 


Introduction 


Imagine you are learning physics while situated in front of your 
computer. Instead of reading terse texts on Newton's First Law, you are 
immersed in a game where you must use the relevant principles of the 
law to solve a level and advance in the game. Does this sound more 
appealing than reading? Now imagine playing a level where you feel 
stuck, unable to find a solution. You grow increasingly confused, fru- 
strated, and possibly angry. You may pause, telling yourself that it is 
just a game and you can figure it out, take a deep breath to calm down, 
attempt a new strategy, and ultimately solve the level. This process of 
affect generation and affect regulation is part and parcel of game-based 
learning (GBL) (Gutica & Conati, 2013; Sabourin & Lester, 2014; Shute, 
D’Mello, et al., 2015 and Shute, Ventura, et al., 2015), and learning 
more generally (Calvo & D'Mello, 2011; Kim & Pekrun, 2014; Pekrun & 
Linnenbrink-Garcia, 2014). Whereas several studies have investigated 
the incidence of affective states in GBL (Gutica & Conati, 2013; 
Sabourin & Lester, 2014; Shute, D’Mello, et al., 2015 and Shute, 


Ventura, et al., 2015), few have investigated affective regulation in GBL, 
which pertains to the set of processes individuals use to increase, de- 
crease, or maintain particular affective states in order to achieve de- 
sired outcomes (Gross, 1998), a gap we address in this paper. 

In doing so we contribute to models of self-regulated learning by 
Zimmerman (1989), Boekaerts (1991), Winne and Hadwin (1998), 
Pintrich and De Groot (1990), and more recently, by Efklides (2011) 
and Jarvela and Hadwin (2013), especially those that espouse im- 
portant roles for affect and its regulation, such as models by Boekaerts 
(1991) and Jarvela and Hadwin (2013). Some of these models 
(e.g.,Boekaerts, 1991; Jarvela & Hadwin, 2013) consider affect reg- 
ulation as an important component of self-regulated learning, but em- 
pirical research on specific affect regulation strategies has been slow to 
emerge (Panadero, 2017). We advance the empirical knowledge base of 
these models by explicitly investigating affect regulation strategies and 
their relationship with the learning process and learning outcomes. 
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1.1. Background 


We situated our work in the context of GBL because, compared to 
less engaging and interactive learning environments where affect reg- 
ulation has been studied (Price, Mudrick, Taub, & Azevedo, 2018; 
Strain & D'Mello, 2014), well-designed games are known to produce 
emotionally-rich learning experiences (see Clark, Tanner-Smith, & 
Killingsworth, 2016 for a meta-analysis of digital games and learning). 
Accordingly, our work connects three research areas — game-based 
learning, affect during learning, and affect regulation; which we briefly 
review below. 

Game-based learning. Game-based learning refers to using well- 
designed games as vehicles to support learning of various competencies, 
attributes, and outcomes such as visual-spatial abilities and attention 
(e.g., Green & Bavelier, 2007, 2012; Shute, Ventura, & Ke, 2015), 
openness to experience (Chory & Goodboy, 2011; Ventura, Shute, & 
Kim, 2012), persistence (Ventura, Shute, & Zhao, 2012), creativity 
(Jackson et al., 2012), civic engagement (Ferguson & Garza, 2011), as 
well as valuable academic content and skills (for reviews, see Tobias & 
Fletcher, 2011; Wilson et al., 2009; Young et al., 2012). Games that 
incorporate problem solving, adaptive challenges, and ongoing feed- 
back can trigger and sustain interest and motivation, in turn supporting 
engagement and learning (e.g., Shute, Rieber, & Van Eck, 2011). In 
addition, adaptive challenges and dynamic performance feedback in a 
game helps create an environment that can foster the sense of flow 
(Csikszentmihalyi, 1990) and potentially cultivate the mindset that 
generates persistence and effort-driven competency development 
(Dweck, 2006; Yeager & Dweck, 2012). Moreover, educational games 
do, in fact, foster learning (Shute, Leighton, Jang, & Chu, 2016), by 
incorporating effective learning principles, such as the provision of 
ongoing feedback, interactivity, and active participation - factors 
known to lead to improvements in knowledge and skill acquisition 
(Gee, 2003; Ifenthaler, Eseryel, & Ge, 2012; Shute, Ke, & Wang, 2017). 

Despite considerable progress in GBL over the last two decades, 
there is still much room for improvement. One of the key goals in the 
design of educational games is to create an engaging and flexible en- 
vironment that supports learning for a broad range of learners. 
Achieving this goal depends on measuring pertinent learner char- 
acteristics—e.g., prior knowledge, affective states, motivation—and 
determining how to use that information to improve the gaming ex- 
perience and learning outcomes (Conati, 2002; Shute, Lajoie, & Gluck, 
2000; Shute & Zapata-Rivera, 2012). Here, we focus on how games can 
support learners by helping them regulate the affective states that they 
will inevitably experience during gameplay. 

Affective States during Learning. From frustration to joy, anxiety 
to curiosity, learners experience a rollercoaster of affective states’ 
during learning, with GBL being no exception (Shute, D’Mello, et al., 
2015 and Shute, Ventura, et al., 2015). Although one strength of well- 
designed educational games is their ability to stimulate positive affec- 
tive states (e.g., delight, eureka), which have the power to prime the 
cognitive system toward heuristic-driven processing and creative ex- 
ploration (Clore & Huntsinger, 2007; Isen, Daubman, & Nowicki, 1987), 
the challenge-inspired nature of well-designed games can also trigger 
negative affective states (e.g., confusion, frustration). Students get 
confused when outcomes do not match expectations, when they en- 
counter challenging impasses, and when they are unsure of how to 
proceed (D'Mello & Graesser, 2014a; VanLehn, Siler, Murray, 
Yamauchi, & Baggett, 2003). Frustration occurs when students make 
mistakes, get stuck, and run out of options on how to resolve obstacles 
that block goals (Kapoor, Burleson, & Picard, 2007; Stein & Levine, 
1991). Negative affective states, however, play an important role in 


1 We use the broad term “affective states” rather than the more restrictive 
term “emotions”, which does not cover the range of states learners experience, 
such as confusion and determination (see D'Mello & Graesser, 2014a). 
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learning (D'Mello & Graesser, 2014b; Kim & Pekrun, 2014) because 
affect is more than just incidental; it is functional. 

Affective states perform signaling functions (Schwarz, 2000), for 
example, by highlighting problems with knowledge (confusion), as well 
as evaluative functions by appraising events in terms of their value, goal 
relevance, and goal congruence (Izard, 2010; Stein & Levine, 1991). 
Affective states also perform modulation functions (reviewed in Fiedler 
& Beier, 2014) by constraining or expanding cognitive focus—as is the 
case when positive affective states facilitate creative problem solving by 
engendering broader, top-down, generative processing (expanded 
focus) (Isen et al., 1987), and when negative affective states trigger 
narrow, bottom-up, and focused modes of processing (constrained 
focus) (Barth & Funke, 2010; Schwarz, 2000). 

But it is misleading to adopt a simplistic view that positive affect is 
beneficial and negative affect is harmful. The mechanisms by which 
affect influences learning depends on multiple factors, such as the level 
of arousal aligned with task demands (Mandler, 1984; Yerkes & 
Dodson, 1908) and the extent to which affective thoughts consume 
working memory resources, thereby increasing cognitive load (Eysenck, 
1985; Fraser et al., 2012; Paas & Ayres, 2014; Ramirez & Beilock, 
2011). Thus, both positive and negative affective states can be bene- 
ficial for learning in certain contexts, such as when happiness leads to 
more creativity (Isen et al., 1987), sadness leads to deeper analytical 
processing (Mills, Wu, & D'Mello, in press), or when feeling somewhat 
anxious can galvanize attentional resources (Pacheco-Unguetti, Acosta, 
Callejas, & Lupianez, 2010). 

There are also situations where affective states can be harmful for 
learning (Gross & Jazaieri, 2014). Whereas some confusion and frus- 
tration are an important part of complex learning (D'Mello & Graesser, 
2014a; Lehman, D'Mello, & Graesser, 2012), intense or prolonged 
confusion and frustration can lead to lower learning outcomes (Pekrun, 
Goetz, Daniels, Stupnisky, & Perry, 2010). Prolonged frustration, for 
example, can lead to anxiety and despair (Zeidner, 2007) and eventual 
disengagement (Pekrun et al., 2010). Instead of engaging deeply in 
creative exploration and knowledge construction, struggling and dis- 
engaged students exhibit problematic behaviors such as systematic 
guessing (Baker, Corbett, Koedinger, & Wagner, 2004) or trying to 
obtain solutions from other students rather than discovering them on 
their own (Nelson-Legall, 1987). Such instances of intense negative 
affective states necessitate the need for strategies to help learners reg- 
ulate these states. 

Affective Regulation. Affective regulation refers to efforts to in- 
fluence which affective states one has, when one has them, and how one 
experiences or expresses them (Gross, 1998). Ranging from attempts to 
think about a situation differently, focusing on one's breathing, 
punching a wall, biting one's nails, or getting on social media for a 
distraction, affective regulation can take a variety of forms. Un- 
surprisingly, different regulation strategies produce different outcomes 
due to their differential effects on physiological and cognitive pro- 
cesses, such as physiological arousal, attentional focus, and cognitive 
load. 

One popular framework for organizing these strategies is the process 
model of emotion regulation (see Gross, 2015 for a review), which frames 
the regulatory process in four distinct strategies (although regulation 
strategies are often used in combination; Werner, Goldin, Ball, 
Heimberg, & Gross, 2011). The first strategy, situation modification,” 
refers to taking steps toward altering a situation to change its emotional 
impact (e.g., getting on social media rather than doing a tedious 
homework assignment to prevent boredom). Although modifying one's 
situation can lead to short-term relief, this strategy is detrimental if it 


? Because situation modification could involve creating a new situation, there 
is not a clear line between situation modification and situation selection (i.e., 
choosing which situation to engage in), though earlier models of affective 
regulation separated the two (Gross, 1998). 
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prevents exposure to a situation which has longer-term benefits (e.g., 
engaging in a tedious skill-building task is beneficial years later, Galla 
et al., 2014). The second affective regulation strategy, attentional de- 
ployment, refers to shifting one's attention either within a given situa- 
tion (e.g., shifting attention from one game feature to another) or 
shifting attention away from the situation altogether (e.g., intentionally 
thinking about what you will eat for dinner while playing the game). It 
might also involve ruminating on the emotional experience, which can 
occupy working memory resources (Curci, Lanciano, Soleti, & Rimé, 
2013). 

The third strategy, cognitive change, refers to modifying one's ap- 
praisal of a situation (e.g., telling oneself that a game is really fun de- 
spite being frustrated). Cognitive reappraisal is the most well-studied 
form of cognitive change, which involves systematically changing one's 
appraisals about a situation in order to alter its affective impact. For 
example, participants instructed to think objectively in order to de- 
crease emotional reactivity to affectively charged films, for example by 
taking the role of a medical professional or by focusing on technical 
aspects of disgust-inducing videos of surgical procedures, vomiting, and 
animal slaughter, experienced less negative affect than participants 
who used no reappraisal strategy (Goldin, McRae, Ramel, & Gross, 
2008). Relatedly, emerging research has shown that cognitive re- 
appraisal can be beneficial in learning contexts, including standardized 
test tasking and reading comprehension (Jamieson, Mendes, 
Blackstock, & Schmader, 2010; Strain & D'Mello, 2014). 

The final strategy, response modulation, refers to directly influencing 
experiential, behavioral, or physiological components of the affective 
response. It can take many forms, one of which is relaxation or deep 
breathing to alter one's physiological responses (Thayer & Lane, 2009). 
One of the most studied forms of response modulation, however, is 
expressive suppression, or inhibiting one's emotion-expressive beha- 
vior. Suppression is generally a maladaptive regulation strategy and can 
lead to impaired memory (Johns, Inzlicht, & Schmader, 2008; Richards, 
Butler, & Gross, 2003) and greater activation in emotion-generative 
brain regions, which can negatively influence cognitive control (Goldin 
et al., 2008). 

Affective regulation influences the quality of an affective state as 
well as its intensity or the time at which it is experienced (Gross, 1998). 
The use of adaptive affective regulation strategies has positive effects 
not only on obvious dimensions, such as subjective well-being, but also 
on academic performance (Gross, 2015; Voltmer & von Salisch, 2017) 
because affective regulation plays a central role in learning and 
memory (e.g., Gross, 2015; Jarvenoja & Jarvela, 2009; McRae, 2016). 
In emotion regulation research, it is common to distinguish adaptive 
from maladaptive emotion regulation strategies depending on their 
associations with psychological well-being. However, this distinction 
might be too simplistic in learning contexts, in which positive and ne- 
gative affective states can differentially impact learning outcomes (see 
above). 


1.2. Current study 


Most educational games promote a high degree of interactivity 
compared with existing alternatives (e.g., reading textbooks, listening 
to lectures). Play involves active cognitive and/or physical engagement 
that allows for the freedom to fail (and recover) and to experiment 
freely (Rieber, 1996). These features also contribute to a rich affective 
experience during gameplay, yet we know relatively little about how 
learners regulate those states and which regulation strategies are ben- 
eficial, harmful, or benign. 

We address this gap by conducting an online study where partici- 
pants played three levels of an educational game called Physics 
Playground (Shute & Ventura, 2013), and, after each level, reported 
their strongest affective state and what they did to regulate that state. 
After completing all levels, we assessed both effort put into gameplay 
(using a self-report survey) and physics knowledge (using a physics 
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posttest). 

Our study was designed to address the following research questions: 
(1) Which affective regulation strategies do learners engage in to 
manage their affective states during short but challenging gameplay 
with Physics Playground? and (2) What is the relationship among af- 
fective states, affect regulation strategies, and outcomes (gameplay 
success and posttest scores) and what factors moderate these relation- 
ships? In addition to advancing theories of self-regulated learning by 
contributing to the knowledge base on affect regulation, addressing 
these questions is also a first step towards a broader goal of developing 
in-game supports to help learners productively regulate their affect to 
promote engagement and learning. 

It is important to consider a number of design decisions that guided 
the study design. Recall that our primary focus is on investigating affect 
regulation and its influence on gameplay outcomes. To keep scope 
manageable, we focused on the “strongest” affective state and the 
regulation strategy used to address it rather than consider multiple 
states or co-occurring affective states as this could quickly lead to a 
combinatorial explosion in possibilities. 

Next, to keep total study time manageable, while maximizing ga- 
meplay time, we did not include a pretest. Thus, rather than examining 
learning gains (i.e., pretest to posttest improvement) as a function of 
gameplay, we focused on the relationships among in-game affective 
states, effort expended in the game, and affect regulation strategies 
relative to posttest scores. 

To increase the likelihood of eliciting strong emotions and asso- 
ciated regulatory strategies, we manipulated level difficulty in that each 
participant completed one easy, one medium, and one difficult level. 
Because the manipulation was intended to induce strong emotional 
responses, it was necessary to limit gameplay to three levels for ethical 
purposes. 

Finally, we wanted to ensure that participants meaningfully en- 
gaged in each level in order to experience a strong emotion and have an 
opportunity to regulate it rather than immediately quit. Therefore, we 
required participants to play each level for a minimum of 4-min before 
they were allowed to quit the level, unless they solved it within 4-min. 
They could resume playing after the initial 4-min had elapsed and many 
opted to do so; there was no further restrictions for the level in that 
participants could either solve the level or quit at any time. We do not 
think that the 4-min minimum of gameplay per level (unless they solved 
it within 4-min) impedes an analysis of affect regulation. Prior research 
has also successfully investigated affect regulation in much shorter time 
intervals (e.g., in as little as 5 min in a recent study on frustration tol- 
erance (Meindl, Yu, Galla, Quirk, Haeck, & Goyer, in press)). 

In summary, these choices were design decisions made to satisfy the 
goals of the present study, but should be considered when applying the 
findings more broadly because, like any study, our findings are con- 
strained within the research context — in this case, an online study in- 
volving three, relatively short, levels of varying difficulty in Physics 
Playground. 


2. Method 
2.1. Participants 


We recruited 125 individuals through TurkPrime (Litman, 
Robinson, & Abberbock, 2016), an Internet-based research platform 
that integrates with Amazon Mechanical Turk (MTurk) where in- 
dividuals complete Human Intelligence Tasks (HITs) for monetary 
compensation. Participation in the study was voluntary and was ap- 
proved by the Institutional Review Board at the first author's university. 
Participants received $4.50 for completing the 30-40-min study. Par- 
ticipation was restricted to college students in the United States based 
on self-reported information on TurkPrime. The sample reduced to 110 
(Mage = 22.14, SDage = 1.24; 50.0% female; 70.0% White) after re- 
moving three participants who completed the study twice and 12 who 
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2.2. Pilot studies 


All materials and procedures were piloted prior to conducting the 
main study using small groups of participants recruited through MTurk. 
The goal of the pilots was to ensure that the technology functioned 
appropriately, the instructions and questions could be easily under- 
stood, relevant affective states were included, and the difficulty ma- 
nipulation was effective. Participants had the option of providing open- 
ended feedback, which we used to make iterative modifications to the 
study. 


2.3. Physics Playground 


Physics Playground’ (Shute & Ventura, 2013) is a 2D educational 
video game designed to enhance learning of qualitative physics prin- 
ciples (Ploetzner & VanLehn, 1997) related to the main concepts of 
Newton's laws of force and motion, linear momentum, energy, and 
torque. The game obeys the basic rules of physics and dynamically 
responds to players' interactions with the game. This responsiveness is 
accomplished via detailed formal simulation of a virtual physics 
“world” using actual, accurate physics formulas and calculations to 
account for mass, gravity, friction, momentum, and many other physics 
concepts. 

The primary goal is for players to guide a green ball to a red balloon, 
resulting in “solving” the level (see Fig. 1). To do so, players must create 
agents—ramps, pendulums, levers, and springboards—that “come to 
life” on the screen. A ramp is any line drawn that helps to guide the ball 
in motion (e.g., to get the ball over a hole). A swinging pendulum exerts 
horizontal force, directing an impulse tangent to its direction of motion. 
Levers rotate around a fixed point while a springboard stores elastic 
potential energy provided by a falling weight, both of which are useful 
for moving the ball vertically. Game elements in Physics Playground 
include ongoing feedback, interactive problem solving, and adaptive 
challenges. The game also gives players the freedom to try/fail, where 
failure in this context is not a bad thing, but instead, provides valuable 
information on how to proceed next. Moreover, because there is not just 
one correct “answer” to a problem in Physics Playground, and the game 
allows players to create their solutions by drawing objects that come 
alive, these features foster curiosity, which is not typically present in 
more traditional learning environments. 

Fig. 1 shows a sample level (medium difficulty — see below) where a 
player draws a pendulum on a pin (little black circle) to make it swing 
down to hit the ball (surrounded by a heavy container hanging from a 
rope). To succeed, the player should manipulate the mass distribution 
of the club (green mass on the right) and the angle from which it was 
dropped to accomplish just the right amount of force to get the ball to 
the balloon. 

Shute, Ventura, & Kim (2013) reported that performance in the 
game significantly correlated with posttest scores. Further, after 4h of 
gameplay with no instruction or any other learning supports, students 
improved in qualitative physics understanding from pretest to posttest 
(Cohen's d = 0.23). These findings have been replicated by Shute, 
D’Mello, et al. (2015) and Shute, Ventura, et al. (2015). 


2.4. Difficulty manipulation 


Physics Playground levels range in difficulty, which is based on a 
number of factors including the relative location of ball to balloon, 
number of obstacles present, number of agents required to solve the 
level, and novelty of the level. Level difficulty was quantified by experts 


3 Please see https://www.youtube.com/watch?v =RRR2vBzCIQQ for a demo 
video. 
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Fig. 1. Example of a pendulum solution in Physics Playground. The dashed 
lines (for illustrative purposes only; not shown in the game) demonstrate the 
trajectory of the pendulum to the ball (green circle) and then up to the balloon 
(target). 


based on the game mechanics (1-5 scale) and complexity of physics 
principles (1-5 scale), with total difficulty scores ranging from 2 to 10. 
We selected 12 (out of 75) levels focused on two physics concepts 
(“Energy can Transfer” and “Properties of Torque”) for the current 
study, and categorized them as easy (3 levels, Mairticuty = 5.33, 
SD = 0.58), medium (6 levels, Maitficulty = 7-00, SD = 0.00), and diffi- 
cult (3 levels, Maifficutty = 8.33, SD = 0.58). Participants were randomly 
assigned one level from each of the three difficulty categories (i.e., each 
participant played one easy, one medium, and one difficult level). 
Everyone started with an easy level, but the order of the medium and 
difficult levels was counterbalanced across participants (easy-medium- 
difficult vs. easy-difficult-medium). 


2.5. Measures 


Perceived Difficulty. After playing each level, participants were 
asked to rate their perceived difficulty of the level (“I believe the level I 
just played was very difficult”) using a scale of 1 = Strongly Disagree to 
7 = Strongly Agree. 

Strongest Affective State. Our goal was to study affect regulation 
strategies corresponding to specific affective states. Accordingly, after 
each level of gameplay, we asked participants to select the strongest 
affective state they experienced while playing that level (“During the 
level you just played, what was the strongest emotion you felt?”). 
Specifically, participants selected one (and only one) out of a list of nine 
options: anxious (stressed, nervous), bored (disinterested), confused, 
curious, happy (accomplished, relieved), determined (engaged, fo- 
cused), frustrated (annoyed, angry), sad (unaccomplished, dis- 
appointed), and other (not listed). We selected these affective terms 
from previous work on affective states during learning with technology 
(D'Mello, 2013), a previous study which tracked affective states during 
Physics Playground (Bosch, D'Mello, Ocumpaugh, Baker, & Shute, 
2016), and small pilots conducted before the main study. 

We focused on a single choice protocol where participants select 
their strongest affective states rather than reporting all affective states 
using a Likert-type scale, or checkboxes where participants would select 
multiple affective states. This is because we were interested in under- 
standing emotion regulation strategies targeted at upregulating or 
downregulating specific affective states (see below). It would be un- 
tenable to ask participants to select strategies aimed at all nine affective 
states, so we elected for the strongest state. However, to obtain addi- 
tional information of affective intensity, after selecting their strongest 
affective state, participants self-reported the strength of that affective 
state (“How strongly did you feel this emotion?“, 1 = Not at all, 5 =A 
great deal). 

We further note that the retrospective self-report method utilized 
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Table 1 
Affective regulation strategies situated within the process model of emotion regulation. 


Steps in Process Model of Emotion Regulation Affective Regulation Strategy Item 


No Regulation Acceptance “Accepted emotion and did not try to change it” 

Situation Modification Situation Modification “Changed my situation, e.g., turned body, moved locations, changed lighting” 

Attentional Deployment Attentional Redirection “Distracted myself. E.g., looked around the room, thought about something unrelated to the 
game” 

Cognitive Change Cognitive Reappraisal “Focused on the game but thought about the situation differently, e.g., told myself things to 
change how I felt” 

Response Modulation Relaxation “Relaxed my body, e.g., took deep breaths, unclenched my jaw, relaxed shoulders” 

Suppression “Hid my emotion, e.g., looked calm so no one could tell I was experiencing an emotion” 


Note. We presented the items in a checkbox format (in a random order after each level) where participants could choose as many emotion strategies as they used. 


here relies on participants' ability to recall their strongest emotion 
during the past few minutes of gameplay. Whereas previous research 
indicates a strong correlation between retrospective and concurrent 
self-reporting, even after a much longer delay (D'Mello & Graesser, 
2014b), there are tradeoffs to both approaches as discussed in a pre- 
vious review (Porayska-Pomsta, Mavrikis, D'Mello, Conati, & Baker, 
2013). We elected for the retrospective approach so as to not interrupt 
participants during gameplay, to not interfere with their affective reg- 
ulation strategies because self-reporting an emotion is likely to be re- 
active, and due the relatively short time span between gameplay and 
affect reporting. 

Affective Regulation Strategies. We assessed affect regulation 
strategies aimed at the strongest affective states with the following 
questions: “What did you do to manage your strongest emotion?” 


An object is hanging on a tree branch. What would a) By making the object lighter 


make the branch more likely to break? b) By moving the object farther from the tree 
Participants selected among six affective regulation strategies (see trunk 
Table 1) using a checkbox that allowed participants to choose multiple c) By moving the object closer to the tree trunk 
strategies. The items were modified from the Emotion Regulation In- Hoare panera penis 
terview (Werner et al., 2011) to specifically refer to gameplay. We Fig. 2. Example test item. The correct answer is “b” because increasing the 
further modified the Emotion Regulation Interview so that participants distance from the fulcrum (i.e., tree trunk) increases the torque. 


simply reported whether they used the strategy rather than the per- 
centage of time they used the strategy. 


Effort. Participants reported the amount of effort they put into ssa [——+| Pal orl [—> ccna 
playing the game (e.g., “I tried very hard in this game”, “It was im- 
portant to me to do well in this game”) using the five-item effort sub- J 
scale (a = 0.85) of the Intrinsic Motivation Inventory (Ryan & Deci, Randomization to Level Order 
2000). Participants provided responses on a scale of 1 = Strongly Dis- 
agree to 7 = Strongly Agree. Easy-Medium-Difficult : Easy-Difficult-Medium 
Gameplay Logs. The game was played online and the log files re- Orden = 60) ; {n=50) 
corded what the participants did in the game (e.g., agents drawn, x * 
whether the level was solved). Here, we focus on whether participants Same Tutotel |) «|, | Gamestutoftal 


successfully solved a level (i.e., in-game success). We did not analyze ‘ | v 


the choice to continue playing a level vs. quitting after the quit option Repeat After Each Level 
was enabled (i.e., after the initial 4-min) because this option is only evel i i Easy Perceived Difficulty 
available to those who did not complete the level within 4-min, thereby a i Strongest Affective State 
further reducing the sample size and introducing a potential confound. SSaS9mananaaaD EE ¥ 
Further, this variable is subsumed by the in-game success variable. Level 2 Medium | Difficult |__| |_Strength of eee State 


Physics Posttest. Participants completed four pictorial multiple- i Cause of Affective State 
i y v 


choice physics questions developed to assess their conceptual under- - - 

7 : H Regulation Strategies 
standing of the selected physics concepts “Energy can Transfer” and Level 3 Difficult Medium 
“Properties of Torque.” The items were patterned after the Force Other Affective States 
Concept Inventory (Hestenes, Wells, & Swackhamer, 1992), which 
comprises a far transfer assessment from qualitative understanding of 
physics in the game to more formal knowledge. For example, Fig. 2 cain Physics 
depicts one item (targeting torque) that shows a picture of a tree with a Questionnaire [~ | Test [| enriet: 
branch on which hangs a weight. The question is: “An object is hanging 
on a tree branch. What would make the branch more likely to break?” 
The four response options include: a) by making the object lighter; b) by 
moving the object farther away from the tree trunk; c) by moving the were then piloted with participants on MTurk prior to administration. 
object closer to the tree trunk; and d) moving the object won't make a 
difference. The correct answer is “b” because increasing the distance 
from the fulcrum (i.e., tree trunk) increases the torque. Two physics 
experts reviewed the tests and provided recommendations. The tests 


Fig. 3. Protocol flowchart. 


2.6. Procedure 
A flowchart of the protocol is shown in Fig. 3. After providing 
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consent, participants completed demographic questions and a short 
personality assessment (not analyzed for the present study due to low 
reliabilities) and then learned how to play the game by completing a 
short 5-min tutorial. Next, participants were randomly assigned to one 
of the difficulty orders (see above). We did not inform participants 
about the difficulty of the levels. Of the 110 participants (Mage = 22.18, 
SDage = 1.21; 38.3% female; 78.3% White) 60 of them completed the 
levels in easy-medium-difficult order and 50 (Mage = 22.08, 
SDage = 1.28; 64.0% female; 60.0% White) were assigned to the easy- 
difficult-medium order. 

Participants completed all measures using Qualtrics on a web 
browser on a PC as the game cannot be accessed via smartphone. 
Participants had 4-min to attempt each level before they were allowed 
to quit, beyond which they were provided with an opportunity to re- 
sume playing the level or to quit and move on. They could of course 
move on whenever they solved the level even within the initial 4-min. 
Next, participants rated the difficulty of the level. Then, participants 
selected the strongest affective state they felt while playing the level, 
rated the strength of that affective state, selected causes of the affective 
state (couldn't solve the level, solved the level, the level was difficult, 
other — not analyzed here),* and the affective regulation strategies they 
used to manage their strongest affective state. Then, participants had 
the option to select any other affective states they experienced while 
playing the level (choosing as many as applied) from the same nine 
options presented to them previously.° The procedure was repeated for 
the second and third levels, upon which participants completed the 
effort questionnaire and the posttest. Participants completed the study 
in an average of 35.2 min (SD = 14.3 min). 


2.7. Data treatment 


Statistical Models. Table 2 shows means and standard deviations 
for all variables analyzed. Due to the repeated, nested, and binary 
structure of the data, we used mixed-effects logistic regression models 
(Pinheiro & Bates, 2000) for all item-level analyses (a game level is an 
item) using the “Ime4” library (Bates, Maechler, Bolker, & Walker, 
2015) in R (R Core Team, 2017). Level difficulty and order were in- 
cluded as fixed effects covariates and participant was included as a 
random intercept. Omnibus effects were analyzed using a Type II Wald 
Chi-Square Test using the “car” package (Fox & Weisberg, 2011). We 
used the emmeans (estimated marginal means) package for posthoc 
comparisons with a False-Discovery Rate (FDR) adjustment (Yekutieli & 
Benjamini, 1999) for multiple comparisons. When physics posttest was 
the outcome, data were analyzed with ordinary least squares (OLS) 
regression models at the participant level. We used two-tailed tests with 
a significance criterion of 0.05 for all analyses. 

Analytic Strategy. We measured a large set of affective states and 
regulation strategies because there was little prior research for gui- 
dance. However, the complexity of the data with nine affective states 
and six affect regulation strategies across three different levels of dif- 
ficulty would yield a total of 162 unique relationships to explore for 
each dependent variable. To reduce complexity and to reduce the 
propensity of incurring Type 1 errors due to the large number of ana- 
lyses as well as Type 2 errors by applying stringent creation for multiple 
comparisons, we opted to focus on the most frequent occurrences and 
strongest effects. We first examined the prevalence of the self-reported 


4We do not include the cause of emotion in the analysis because the re- 
sponses were synonymous with solving a level (e.g., I solved the level, I couldn't 
solve the level). Additionally, almost all respondents chose “The level was 
difficult.” 

° These data were collected as a further exploratory aim on co-occurring af- 
fective states and associated affect regulation strategies. We do not include 
them in our primary analyses because we do not have affect regulation stra- 
tegies associated for these additional states and because our focus is on the 
strongest affective states. 
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strongest affective states (anxious, bored, confused, curious, de- 
termined, frustrated, happy, or sad) after each level (Fig. 4). Across the 
three levels of play, participants primarily reported determination or 
frustration as their strongest affective state. Curiosity occurs when an 
individual detects an impasse but perceive that it can be resolved if they 
are sufficiently determined to do so (Berlyne, 1978; Loewenstein, 1994; 
Silvia, 2010). Curiosity is also an important component of intrinsic 
motivation that is central to self-determination theory (Deci & Ryan, 
1985). Thus, we created one variable that reflected determination or 
curiosity (note the “or” instead of the “and” — i.e., we are not treating 
them as an affective blend). Similarly, confusion and frustration are 
closely linked to the process of impasse detection and resolution in that 
confusion signals an impasse and frustration occurs when repeated at- 
tempts to resolve the impasse fail, thereby reducing coping potential 
(D'Mello & Graesser, 2012; D'Mello, Lehman, Pekrun, & Graesser, 2014; 
D'Mello & Graesser, 2014b; Meindl et al., in press). Some initial re- 
search also suggests that frustration and confusion co-occur during 
learning across multiple contexts including expert tutoring, learning 
from MOOCs, and learning computer programming (Bosch & D'Mello, 
2014; Dillon et al., 2016; Lehman, D'Mello, & Person, 2010) and often 
precede each other across short time internals of 15-20s (Bosch & 
D'Mello, 2017; D'Mello & Graesser, 2012). Hence, we created another 
variable called frustration/confusion to reflect the occurrence of either 
frustration or confusion. 

Next, because the other affective states (anxious, bored, happy, and 
sad) individually occurred less than 10% of the time across levels, we 
only focused on these two sets of states. This required us to remove the 
two participants (< 2%) who never reported determination/curiosity 
or frustration/confusion states during gameplay. Subsequent analyses 
include 82.4% of the levels (267 observations out of 324, n = 108) 
when either of these states were reported. Because we focused on the 
strongest affective state per level, for analyses conducted at the item- 
level (i.e., a game level), this could either be determination/curiosity or 
frustration/confusion, but never both. And because there was no dif- 
ference in self-reported strength of determination/curiosity (M = 3.76, 
SD = 0.83) compared to frustration/confusion (M = 3.90, SD = 0.93), 
p = .89, for parsimony,° we created one binary variable reflecting the 
strongest affective state - determination/curiosity (reference group) vs. 
frustration/confusion. We note that this was only done for the item- 
level analyses; the two variables are treated separately for participant- 
level analyses. 

Turning to affect regulation strategies, participants primarily re- 
ported using cognitive reappraisal or acceptance (i.e., no regulation) 
(Fig. 5). Because participants rarely (< 5%) reported other regulation 
strategies (situation modification, attentional redirection, relaxation, 
and suppression), we focused only on cognitive reappraisal and ac- 
ceptance. These two strategies are treated independently because par- 
ticipants could engage in either of them or in both. Thus, the sub- 
sequent analyses focus on associations between frustration/confusion 
(vs. determination/curiosity) and the affective regulation strategies of 
cognitive reappraisal and acceptance. 

Level Difficulty Manipulation Check. We regressed perceived 
difficulty on actual level difficulty (easy, medium, difficult), order 
(easy-medium-difficult or easy-difficult-medium), and the level diffi- 
culty x order interaction. As expected, there was a significant effect of 
actual difficulty on perceived difficulty, x72, N = 267) = 34.1, 
p < .001. Pairwise comparisons using an FDR correction revealed that 
participants' difficulty ratings were higher for the more challenging 
levels: difficult > medium > easy. There was no effect of order, y°(1, 
N = 267) = 1.29, p = .26, and no difficulty x order interaction, y7(2, 
N = 267) = 0.16, p = .92. Thus, the difficulty manipulation was suc- 
cessful. 


© When analyzed separately, effects for one state would simply be (sign) re- 
versed versions of the other, so it would be redundant to report both. 
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Table 2 
Means (with standard deviations in parentheses) for all variables. 
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Measures Item-level analyses 


Participant-level analyses 


Easy Level Medium Level Difficult Level N = 108 
Manipulation Check Range [1-7 Range [1-7 Range [1-7 
Perceived Difficulty 5.08 (1.88) 5.47 (1.77) 6.43 (0.82) - 
Gameplay Range [0-1 Range [0-1 Range [0-1 
Resumed Level 0.39 (0.49) 0.38 (0.49) 0.28 (0.45) - 
Solved Level 0.46 (0.50) 0.24 (0.43) 0.04 (0.20) - 
Strongest Affective State Range [0-1 Range [0-1 Range [0-1 Range [0-3] 
Determination/Curiosity 0.66 (0.48) 0.38 (0.49) 0.25 (0.43) 1.05 (0.91) 
Frustration/Confusion 0.34 (0.48) 0.62 (0.59) 0.75 (0.43) 1.38 (0.96) 
Regulation Strategy Range [0-1 Range [0-1 Range [0-1 Range [0-3] 
Cognitive Reappraisal 0.73 (0.45) 0.45 (0.50) 0.45 (0.50) 1.49 (0.96) 
Acceptance 0.38 (0.49) 0.49 (0.50) 0.54 (0.50) 1.44 (1.05) 
Effort [1-7] - - - 5.96 (0.93) 
Physics Posttest [0-1] - - - 0.48 (0.27) 
| on affective state, (y°(1, N = 267) = 0.61, p = .44). 
To investigate specificity of regulatory strategy to affective state, we 

io regressed cognitive reappraisal (1 [used] or 0) on strongest affective 
ey state controlling for level difficulty and order (Model 2 in Table 3). 
H Level Difficulty Strongest affective state was unrelated to cognitive reappraisal (y7(1, 
8 sid] Easy N = 267) = 0.09, p = .77), but level difficulty significantly predicted 
= Baiainined 4 ae use of cognitive reappraisal OC, N = 267) = 16.9, p < .001) with 
8 ificu 
2 pairwise comparisons revealing the following pattern in the data: 
oo easy > [medium = difficult]. There was no effect of order on cognitive 
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Fig. 4. Percentage of self-reported strongest affective states for each level of 
gameplay. Participants could choose only one affective state and primarily re- 
ported being determined or frustrated. 


Acceptance ~ 


Relaxation - 


> 

a 

2 Situation Modification - 

£ 

5 , 

c Level Difficulty 

‘S Attentional Redirection - 

& Ml easy 

FA BB edu 
Cognitive Reappraisal ~ 

a Difficult 

° B 

3 

= 

< 


Suppression ~ 

30 
Percentage 
Fig. 5. Percentage of self-reported affective regulation strategies for each level 
of gameplay. Participants could select as many strategies as they used to 
manage their strongest affective state. Participants primarily reported cognitive 
reappraisal and acceptance (no regulation). 


3. Results 


3.1. Research question 1: which affective regulation strategies do learners 
engage in? 


Focusing on the strongest affective state (frustration/confusion vs. 
determination/curiosity), we first examined how it was influenced by 
level of difficulty and order (a potential confound). As shown in Model 
1 in Table 3, there was a significant effect of level difficulty (~°(2, 
N = 267) = 28.1, p < .001), with pairwise comparisons indicating 
that participants were significantly (ps < .04) more likely to report 
frustration/confusion as their strongest emotion as levels increased in 
difficulty (difficult > medium > easy). There was no effect of order 
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reappraisal 7d, N = 267) = 0.76, p = .38). Similarly, neither stron- 
gest affective state (y°(1, N = 267) = 0.12, p = .73) nor order (y7(1, 
N = 267) = 0.002, p = .96) predicted use of acceptance (the model did 
not converge with level difficulty included). Thus, participants were 
equally likely to reappraise or accept their strongest affective state but 
were more likely to engage in reappraisal for the easier levels. 


3.2. 


Research Question 2. What is the Relationship among Affect 
Regulation, Gameplay Success, and Posttest Scores, and what are the 
Moderation Factors? 

In-game Success. We investigated whether strongest affective state, 
cognitive reappraisal, and their interactions predicted in-game success 
(i.e., solved level or not; Model 5 in Table 3). There was no effect of 
order on solving a level 7a, N = 267) = 0.03, p = .87), but, as ex- 
pected, participants were significantly (%7(2, N = 267) = 15.8, 
p < .001) more likely to solve the easier levels compared to the 
medium or difficult level, which were on par with each other 
(easy > [medium = difficult]). Importantly, there was a main effect of 
strongest affective state on solving a level (y7(1, N = 267) = 13.9, 
p < .001), meaning that participants were less likely to solve a level 
when their strongest affective state was frustration/confusion com- 
pared to determination/curiosity. There was no main effect of cognitive 
reappraisal (y?(1, N = 267) = 1.1, p = .29), but there was a marginal 
interaction effect between strongest affective state and cognitive re- 
appraisal (y°(1, N = 267) = 3.1, p = .077) (Fig. 6). Specifically, there 
was no effect of cognitive reappraisal on solving a level when the 
strongest affective state was determination/curiosity (p = .92), but 
those who engaged in cognitive reappraisal when frustrated/confused 
were more likely to solve the level (p = .04). 

A similar analysis (see Model 5 in Table 3) with acceptance as the 
emotion regulation strategy did not yield a main effect of acceptance 
7d, N = 267) = 0.18, p = .67) nor an interaction between strongest 
affective state and acceptance QP, N = 267) = 2.53, p = .11). There 
was, however, a main effect of strongest affective state in that partici- 
pants were more likely to solve a level if they were determined/curious 
rather than frustrated/confused ed, N = 267) = 14.3, p < .001). 
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Model coefficients (with standard errors in parentheses) and model fit for logistic mixed effects models. 


Predictors Strongest Affective State 


Affective Regulation 


In-Game Success 


Model 1 


Model 2 Model 3 


Model 4 Model 5 


Frustrated/Confused vs. 


Used Cognitive Used Acceptance [1 


Solved Level [1 Solved Level [1 vs. 


Determined/Curious Reappraisal [1 vs. 0] vs. 0] vs. 0] 0) 
Observations (n) 267 (n = 108) 267 (n = 108) 267 (n = 108) 267 (n = 108) 267 (n = 108) 
Intercept — 0.746 (.334)* 1.294 (.365)* —0.116 (.306) 0.271 (.486) 0.254 (.407) 
Covariates 
Level Difficulty 
Medium vs. Easy 1.519 (.401)* — 1.460 (.394)* - —0.496 (.415) —0.641 (.408) 
Difficult vs. Easy 2.314 (.443)* — 1.463 (.395)* - — 2.507 (.631)* — 2.621 (.634)* 
Order 
Easy-Difficult-Medium vs. Easy-Medium- — 0.294 (.377) 0.297 (.342) —0.018 (.357) — 0.066 (.412) —0.116 (.406) 
Difficult 
Strongest Affective State Frustrated/Confused 0.099 (.334) —0.108 (.312) —3.107 (.899)* — 1.362 (.523)* 
vs. Determined/Curious 
Affective Regulation 
Cognitive Reappraisal 0.047 (.494) 
Acceptance 0.252 (.487) 
Interactions 
Strongest Affective State x Cognitive Reappraisal 1.713 (.969)+ 
Strongest Affective State x Acceptance — 1.412 (.887) 
Model fit 
Marginal/Conditional R? 0.175/0.410 0.104/0.299 0.001/0.275 0.482/0.589 0.443/0.557 


Note. “p < .05. +p < .10. For mixed effects regression models, marginal and conditional R? pertain to variance explained by fixed vs. fixed plus random effects, 


respectively. 


CognitiveReappraisal 


bia Not Uso 


+t Used 


Log Odds of Solving Level 


Determined/Curious Frustrated/Confused 


Strongest Affective State 


Fig. 6. The interaction effect between strongest affective state and cognitive. 
reappraisal on whether or not participants solved a level. Cognitive reappraisal 
had a significant effect on solving only when participants were frustrated/ 
confused. 


Table 4 


Again, participants were more likely to solve an easy level (p < .001, 
easy > [medium = difficult]) and there was no effect of order O?d, 
N = 267) = 0.08, p = .78). 

Posttest Scores. We analyzed physics posttest scores at the parti- 
cipant level by first creating total scores for strongest affective state 
(determination/curiosity and frustration/confusion) and affective reg- 
ulation strategy (cognitive reappraisal and acceptance) by adding the 
number of times participants reported the respective items across the 
three levels (possible score 0 to 3 for each variable). The regression 
models included the same participants (N = 108) as the mixed effects 
models (i.e., participants needed to report determination/curiosity or 
frustration/confusion at least once). Determination/curiosity and frus- 
tration/confusion were highly negatively correlated, r (106) = —0.76, 
p < .01, so we only report models for frustration/confusion. We also 
included perceived effort as a potential moderating factor on the re- 
lationship between affective state, regulation strategy, and posttest 
scores. Level of difficulty is no longer relevant at the participant level 
because of the within-subjects design and order was excluded since it 
was not significant in any of the above analyses. 


Model coefficients (with standard errors in parentheses) and model fit for ordinary least squares regression models (n = 108). 


Predictors Physics Posttest Scores 
Model 1 (w. Reappraisal) Model 2 (w. Acceptance) 

Intercept — 1.306 (.483)* 0.226 (.483) 
Effort 0.310 (.086)* 0.028 (.080) 
Affective State 

Frustration/Confusion 1.176 (.338)* —0.312 (.289) 
Regulation Strategy 

Cognitive Reappraisal 0.655 (.318)* - 

Acceptance - — 0.464 (.288) 
Interactions 

Effort x Frustration/Confusion — 0.207 (.060)* 0.057 (.047) 

Effort x Regulation — 0.113 (.053)* 0.087 (.048) + 

Frustration/Confusion x Regulation — 0.560 (.211)* 0.455 (.155)* 

Effort x Frustration/Confusion x Regulation 0.097 (.035)* —0.080 (.026)* 
Model Fit (Adj. R?) .098 .131 


Note. “p < .05. +p < .10. 
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Fig. 7. The interaction effects of effort, total frustration, and affective regulation strategy. on physics posttest scores. A) Cognitive reappraisal was beneficial for 
posttest scores when participants were both high in effort and high in frustration. B) Acceptance was detrimental to posttest scores when participants were both high 
in effort and high in frustration, but was beneficial when participants were low in effort, yet highly frustrated or high in effort but low in frustration. 


First, we regressed physics posttest scores on effort, total frustra- 
tion/confusion (score 0 to 3), and total cognitive reappraisal (score 0 to 
3), including all two-way and three-way interactions (Table 4, Model 
1). We focus on the interaction in lieu of the main effects because 
significant interactions suggesting main effects may be misleading 
(Maxwell & Delaney, 2000). Effort, frustration/confusion, and cognitive 
reappraisal interacted to predict posttest scores (p = .006) (Fig. 7a). A 
simple slopes analysis revealed that the positive effect of cognitive re- 
appraisal on posttest scores was only significant (B = 0.12, SE = 0.05, 
p = .03) for participants who put a high amount of effort into the game 
(+1 SD above mean) while also being highly frustrated/confused (+1 
SD above mean). There were no other significant slopes for cognitive 
reappraisal on posttest scores (ps > .16). 

Second, we regressed physics posttest scores on effort, total frus- 
tration/confusion, and total acceptance, including their interactions 
(Table 4, Model 2). Again, effort, frustration/confusion, and acceptance 
interacted to predict posttest scores (p = .008) (Fig. 7b). In contrast to 
engaging in cognitive reappraisal (see above), we found that acceptance 
had a marginal negative effect (B 0.10, SE = 0.05, p = .06) on 
posttest scores during high effort and high frustration/confusion. 
However, acceptance had a positive effect on posttest scores when 
participants were either highly frustrated/confused but low in effort 
(B = 0.09, SE = 0.04, p = .03) or high in effort but low in frustration/ 
confusion (B = 0.09, SE = 0.05, p = .07 - marginal). 
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4. Discussion 


We investigated affect and affect regulation during gameplay with 
dual goals of expanding the self-regulation learning literature to include 
more research on affect regulation and on leveraging basic insights 
towards the design of game-based affective learning supports. Our first 
research question pertained to identifying the affect regulation strate- 
gies that learners engage in while playing Physics Playground in the 
context of the present study. We found that participants primarily ex- 
perienced determination/curiosity or frustration/confusion in our game 
(the other affective states occurred less than 5% of the time) and that 
these affective states increased and decreased, respectively, in con- 
junction with game difficulty. We also found that cognitive reappraisal 
and acceptance were strategies participants reported using to regulate 
their affect whereas the others (e.g., attentional redirection, suppres- 
sion) were exceedingly rare. Importantly, participants were more likely 
to use cognitive reappraisal for the easy compared medium and difficult 
levels, but its use was not systematically related to determination/ 
curiosity vs. frustration/confusion. 

Our second research question focused on the relationship between 
affect regulation and both in-game success and posttest scores and on 
factors that moderate these relationships. We found that cognitive re- 
appraisal predicted successful gameplay when participants were fru- 
strated/confused, but not when they were determined/curious. We also 
found that cognitive reappraisal positively predicted posttest scores 
when participants were frustrated/confused throughout gameplay, but 
only for those who reported putting a high amount of effort into the 
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game. Conversely, participants benefitted (in terms of posttest scores) 
from using an acceptance strategy when either low in effort but high in 
frustration/confusion or low in frustration/confusion but high in effort; 
however, acceptance hurt posttest scores when both effort and frus- 
tration/confusion were high. 

Our findings are consistent with contemporary theories on affect 
and affect regulation during cognitive processing. Intense negative af- 
fective states like frustration can undermine higher order cognitive 
processing skills like attention and planning (see Diamond, 2013 for a 
review). Moreover, inhibiting negative affect (e.g., not engaging in af- 
fective regulation when highly frustrated), requires cognitive effort 
(Pennebaker, 1997) and can lead to sustained heighted activity in af- 
fect-generative brain regions compared to regulating those states, and 
this elevated emotional responding can disrupt cognitive processing 
(Lieberman et al., 2007). Results from our exploratory study are con- 
sistent with this notion in that failing to regulate high levels of frus- 
tration/confusion predicted lower gameplay success and posttest phy- 
sics scores. Importantly, engaging in affective regulation, particularly 
using cognitive reappraisal, may have buffered against the negative 
effects of high frustration/confusion on posttest scores, but only when 
accompanied with effort. 

In general, our findings on the facilitative effect of cognitive re- 
appraisal are in line with the process model of emotion regulation, 
which states that cognitive reappraisal is useful for reducing the impact 
of negative affect on various outcomes (Gross, 2015). Importantly, 
however, when it comes to learning outcomes, we found that cognitive 
reappraisal (vs. acceptance) only benefitted those who experienced 
high frustration/confusion and who reported putting considerable ef- 
fort into the task. We interpret these findings through the lens of cog- 
nitive load theory (van Merriénboer & Sweller, 2005) and the Yerkes- 
Dodson Law of optimal arousal (Yerkes & Dodson, 1908). Participants 
who put substantial effort into gameplay likely had high cognitive load. 
If one experiences intense frustration with no regulation strategy to 
lessen its impact, one presumably has fewer cognitive resources to de- 
vote to gameplay, because affect consumes working memory resources 
and because the intense arousal generated from high frustration/con- 
fusion can lead to lower posttest scores. 

However, those who were not highly motivated to succeed during 
gameplay (i.e., low effort) or those who were motivated but were not 
highly frustrated/confused, would not have as high of cognitive load. 
For these individuals, simply accepting their frustration/confusion was 
more beneficial for posttest scores than attempting to regulate it. There 
are two possible explanations for this finding. One hypothesis is that 
individuals who expended lower effort also had lower arousal, hence, 
the experience of high frustration might have increased their arousal to 
a point that was beneficial for cognitive processes — i.e., towards 
moderate levels of arousal as per Yerkes-Dodson. Others have shown 
that negative affective states can indeed benefit learning (D'Mello & 
Graesser, 2014a; Fiedler & Beier, 2014; Kim & Pekrun, 2014; Mills 
et al., in press), which is (partly) likely due to moderate increases in 
arousal ultimately leading to better performance (Yerkes & Dodson, 
1908). Alternatively, it might be the case that frustration/confusion are 
qualitatively different and also differentially experienced when parti- 
cipants are exerting low vs. high effort akin to research on differences in 
boredom and its effects in underwhelming vs. overwhelming situations 
(Acee et al., 2010; Pekrun et al., 2010). If so, then different cognitive 
reappraisal strategies might have been used, resulting in different 
outcomes. Future research is needed to adjudicate among these and 
other possibilities. 

Our results have implications for the design of next-generation 
educational games. Whereas learners might be successful on their own 
at regulating affective states in some circumstances, they might need 
interventions or supports to help them in others. But game-based af- 
fective regulation supports are virtually non-existent in the GBL lit- 
erature, although some work in this direction is emerging (Sabourin & 
Lester, 2014). Given that affective regulation was linked to both game- 
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play success and posttest scores in our study, there is a convincing need 
to design and test interventions aimed at influencing affective regula- 
tion processes in favorable directions. That said, our results show that 
one cannot simply implement cognitive reappraisal interventions to all 
learners without considering the underlying affective states being 
regulated and expended effort. The challenge for researchers and de- 
signers of GBL environments, then, is to determine the appropriate level 
of frustration/confusion for different types of learners, and investigate 
whether targeted cognitive reappraisal strategies may help them 
maintain engagement and learn. 

Like all studies, ours has limitations. First, we could not elucidate 
causal links between affective states, affective regulation, and gameplay 
outcomes due to the correlational nature of the design, which always 
raises the possibility of third variable confounds. Although we did 
manipulate difficulty and explored its effect on resultant affective states 
and regulation strategies, future work should focus on directly manip- 
ulating other key variables such as inducing specific affective states 
and/or instructing participants to engage in particular regulatory stra- 
tegies (e.g., Sheppes & Meiran, 2007; Strain & D'Mello, 2014). Re- 
latedly, the current study was an exploratory study of affective states, 
affective regulation, and effort. We did not specifically predict any of 
the interactions, which should be replicated in future studies with more 
sensitive measures (e.g., see below). 

Second, requiring participants to spend at least 4-min per level 
(unless they solved it within 4-min) and selection of specific levels may 
have influenced affect and regulation strategies. In a similar vein, a 
different prevalence of affective states and/or strategies to regulate 
them may appear in other games or even in other levels within the 
current game. Similarly, the overall short duration of gameplay limited 
to three levels does reduce the “gameness” of the experience. Hence, 
replication in more open-ended game environments for longer periods 
of game-play is warranted. 

Third, we opted to focus our analyses on the strongest affective 
states and the most frequent regulation strategies in order to manage 
the complexity of the potential associations we could explore. We think 
that this decision was well motivated for this early stage of research 
(see Methods). Nevertheless, as a consequence, additional weaker, but 
potentially interesting effects, have yet to be explored. For example, an 
analysis of co-occurring affective states and/or multiple regulation 
strategies might yield additional insights. Because exploring these and 
more complex relationships would require much larger samples, future 
work should consider considerably ramping up the sample sizes of si- 
milar studies. 

Fourth, the online nature of the study using a convenience sample 
via MTurk did pose some limitations with respect to the types of data 
we could collect. We could not collect extensive individual difference 
measures, such as pretest scores, interest in Physics, and prior experi- 
ence with gaming, due to time limits, and had to rely on self-reports of 
inclusion criterion rather than considering verifiable information. We 
also did not record physiology, which would be needed to address some 
of the questions pertaining to optimal levels of arousal for gameplay 
success. Thus, future studies should also focus on more background 
measures as well as measures of physiological arousal. 

Lastly, unlike studies using cognitive reappraisal interventions (e.g., 
Strain & D'Mello, 2014) where participants are given specific instruc- 
tions on how to modify their appraisals of situations, we cannot be sure 
of the specific reappraisal strategies participants used during gameplay. 
Future studies of self-reported affective regulation strategies should 
include follow-up questions to probe the exact strategies players use to 
self-regulate. This data could be used to develop game-specific affective 
regulation strategies for future investigation. 

In summary, well-designed educational games can be intrinsically 
motivating and engaging (Fullerton, Swain, & Hoffman, 2008; Malone 
& Lepper, 1987; Shute et al., 2011) and can lead to a range of positive 
and negative emotional experiences (e.g., Shute, D’Mello, et al., 2015 
and Shute, Ventura, et al., 2015). Hence, interventions aimed at 
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influencing the intensity and duration of affective states in game-based 
settings via targeted affective regulation interventions may be parti- 
cularly useful. The current study took a step in this direction by iden- 
tifying the affective regulation strategies that learners spontaneously 
utilize during game-based learning and investigating how these stra- 
tegies interact with affective states and effort to influence posttest 
scores. This next step is to leverage these insights to develop and test 
game-based supports that help learners manage their affect and ulti- 
mately benefit from the “hard fun” of learning from well-designed 
educational games while simultaneously enhancing our understanding 
of affect regulation, an important component of self-regulation 
learning. 
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